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Abstract. In this paper we study representations of the indefinite orthogonal group 
0(n, m) which are obtained from local theta lifts of one dimensional characters or unitary 
lowest weight modules of the double covers of the symplectic groups. We will compute 
the associated cycles of these representations. We will show that some of these represen- 
tations satisfy a K-type formula in a conjecture of Vogan on unipotent representations. 



1. Introduction 

In this paper we study representations of the indefinite orthogonal group 0(n, m) which 
are obtained from local theta lifts of one dimensional characters or unitary lowest weight 
modules of the double covers of the symplectic groups. We will compute the associated 
cycles of these representations. We will show that some of these representations satisfy a 
.fT-type formula in a conjecture of Vogan on unipotent representations in |V3j . 

First we introduce the representations $ and Let Sp(p(n + m),R) be the meta- 

plectic double cover of Sp(p(n +m), R). It contains a dual pair Sp(p, R)' • 0(n, m). Here 
Sp(p, R)' and 0(n, m) are (possibly split) two covers of Sp(p, R) and 0(n, m) respectively. 
A Harish- Chandra module of Sp(j>, R)' (resp. 0(n, m)) is called a genuine representation 
if it does not factor through the linear group Sp(p, R) (resp. 0(n,m)). 

We will always assume that Sp(p, R)' splits over Sp(p, R). This happens if and only if 
m + n is even. Let (f denote the genuine one dimensional character of Sp(p, R)'. We fix 
an oscillator representation of Sp(p(n + m), R) and we let 0(q') denote the local theta lift 
of q' to 0(n, m) with respect to this oscillator representation [H2J. The module 9(q') is an 
irreducible Harish-Chandra module of 0(n, m). The group 0(n, m) splits over 0(n, m) 
if and only if p is even. Since m + n is even, 0(n, m) has 8 connected components and 
it has four genuine onex dimensional characters. We will choose a genuine character q of 
0(n,m) so that d = ^9{q') is an irreducible Harish-Chandra module of 0(n, m) and has 
a .fT-type decomposition as in (J5J). 

Irreducible representations of O(r') are parameterized by certain arrays of nonnegative 
integers /x = (fix, . . . , /i r /) G Z r . See Section I2TT1 Let r be an irreducible finite dimensional 
representation of O(r') corresponding to \i. Then the local theta lift L{r) of r to a double 
cover Sp(p, R)" is a unitarizable lowest weight module. A result of |EHWj states that 
conversely, a unitarizable lowest weight module of the connected component of a double 
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cover of Sp(p, R) is the restriction of a unique L(t). Let 9(L(r)) denote its local theta 
lift to 0(n,r). We suppose that it is nonzero. We will choose a genuine character $i of 
0(n, r) such that $(r) := q\9(L(r)) is an irreducible Harish-Chandra module of 0(n, r) 
and has a if-type decomposition as in ([7]). We also denote i?(r) by $(//). 

1.1. Throughout this paper, we always assume that n > 2p, m = r + r' > 2p and n + m 
is even. We exclude n = m = 2p. If this happens, then we say that (p, n, m) is in the 
stable range. In Section 12. 3\ we will address the situation when we are outside of this 
range. 

For the rest of this introduction, all groups are complex algebraic groups and all Lie 
algebras are complex Lie algebras. Let K n ^ m = 0(n, C) x 0(m,C). Let (g,K) either be 
(so(n + m, C), i£n,m) or (so(n + r,C),K njr ). Let (71", V^) be the (g, i^)-module ($,V$) or 
(i?(^),V^) respectively. In this paper we will compute the associated variety and cycle 



Let g = t © p denote the complexified Cartan decomposition. We will denote t by t n>m 
and p by p ni?n if = so(n + m, C). Let S(p) = Sym(p) denote the symmetric algebra of p. 
We will choose a finite dimensional i^-invariant subspace Go of V n in Section [3 If n = 
or if 7r = 9(fx) and n < m, then G is the minimal K-type. 

Let {£7 S : s G N} denote the standard filtration of U (g) and let G s = n(U s )G be the K- 
invariant subspace of V n . Then {G s : s G N} is a good filtration of Kr- The graded module 
M = GrVx — (B S £n(G s /G s -i) is an (S(p), i^)-module. Let I denote the annihilator ideal 
of M in S(p). The ideal I cuts out a variety Z in p* called the associated variety. This 
variety is independent of the choice of Go and the good filtration. 

Nilpotent i^m-orbits on p* = p* m are classified by signed Young diagram with n plus 
signs and m minus signs. See Theorem 9.3.4 in [CM] . Every row has pluses and minuses 
signs alternate across it. We will denote a row as a + (resp. a_) if it has a boxes and 
whose first box on the left begins with a plus (resp. minus) sign. 

Proposition 1.2. Suppose (p,n,m) is in stable range. Let O denote the nilpotent K n>rn - 
orbit 2 p + 2 p _l 1 ]^ 2p l m _~ 2p in p*. Then the associated variety of d is the Zariski closure of O. 

1.3. Next we set it = 9 (fx). We assume that (p, n, m) is in stable range and m = r + r'. 
We will define a i^„ ir -orbit 0\ in p* r . We divide into three cases: 



Proposition 1.4. Suppose (p, n, m) is in stable range and m = r+r' . Then the associated 
variety of d(fx) is the Zariski closure of the nilpotent K n ^ r -orbit 0\. 

The proof is given in Section 17.21 

In both Propositions 11.21 and ll.4[ we will also show that the annihilator ideal I is the 
prime ideal in S(p) consisting of functions vanishing on the Zariski closure of the associated 
variety. 

The definition of associated cycles of a Harish-Chandra module is more involved and we 
refer the reader to [V3] for details. Fortunately, by the above propositions, the associated 



of (tt,K). 



(I) If r, r' > p, then we set 0\ = 3+1™ 
(II) If r > p > r', then we set 0\ = 3+2 : 
(III) If r' > p > r, then we set 1 = 3+1 
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variety of it is the closure of a single orbit X in p*. Let x G X and let K x denote 
its stabilizer in K. We have X = Kx ~ K/K x . Let denote the maximal ideal 
of x in S(p). Let TZ(X) denote the ring of regular functions on X. Then the graded 
module M is an (TZ(X), i^)-module. Let M x denote the localization of M at x. We define 
M x := M x /m x M x which is a C-vector space representation of K x . Let x{ x -> M) denote 
its virtual character and let mx — dime M x be its dimension. We say that mx[X] is the 
associated cycle of ir. The definition of the associated cycle is independent of the choice 
of a good filtration and the point x G X. 

Let A4 be the coherent sheaf on X defined by the finitely generated 7?.(A')-module M. 
Then Ai\x is the iT-equivariant locally free sheaf (or vector bundle) on X corresponding 
to the representation M x of K x . A main objective of this paper is to understand the 
sheaf M. 

First we treat the case (tt, K) = Ve), K = K n ^ m = 0(n) x O(m) and X = O. We 
will pick an e G O. Its stabilizer is 

(1) K e - l^omi, ( 5 ) o 2 n 2 ) EP p xP p . ^ & Q(m _ £ ^ | 

where P p = (GL(p) x 0(n — 2p)) x N p is a maximal parabolic subgroup of O(n) stabilizing 
a dimensional isotropic subspace in the standard representation C n . Similarly P p = 
(GL(p) x 0(m — 2p)) x iV' is a maximal parabolic subgroup of O(m). 
Let dp : P p — > C x be the character given by 

(2) ^{goxnx) = det(g). 

We extend this character trivially across O(m). We state our first result. 

ra — m 

Theorem 1.5. (i) We have M e = d p 2 \x e as a representation of K e , and Ai\o is the 
corresponding K-equivariant invertible sheaf on O . 
(ii) The associated cycle of d is 1[0]. 
(Hi) As a K -module 

n-m 

V & = M = M(0) = Ind%d p 2 \ Ke . 

In (iii), Ai(0) denotes the space of the sections of M. over O. The proof of the theorem 
will be given in Section 

We note that (ii) and Proposition 11.21 are immediate consequences of (i). The theorem 
is not entirely new. Part (ii) is a result of |NOZ] . If n = 2p, then (ii) is also a result of [T] . 

n-m 

The equality V# = lnd K d p 2 in (iii) is part of Yang's thesis [Yhj. Our contribution is to 
recognize and construct a natural (9-morphism from Ai to the i^-equivariant invertible 
sheaf on O in (i). We add that such a morphism is implicit in [NOZJ. The merit of 
using (i) is that we could now bypass the ii'-types or Hilbert polynomials computations 
in the previous work alluded above and prove Theorem II. 5( ii) and (iii) more conceptually 
and efficiently We remark that Yamashita has used the concept of coherent sheaves to 
compute the associated cycles of unitary lowest weight modules [Yaj . 
Theorem 11.51 is a necessary step in our main Theorem 11.61 below. 
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Next we treat the case (vr, V n ) = (#(//), V^), K = K n , r = O(n) X O(r) and X = O x . We 
have to impose addition conditions in (III) of Section 11.31 We will denote this sub-cases 
as (IHb) and (Ilia). 

(Ilia) Suppose r' > p > r and n > m. Then we set /i = ^™l p _ r = (^, . . . , ^p, 0, . . . , 0) 

in W' where there are p — r copies of nz ^ nL . See Section [2TT1 for notation. 
(IIIb) Suppose r' > p > r and n < m. Then we set p — (0, . . . , 0) e W' . 

We pick an e± G Ox- It is more complicated to describe its stabilizer K\ = K ei and 
its representation M ei . We will do this carefully in Sections 17.31 and 17.61 where we will 
define a representation Mi of K\ such that M ei = M\ for some many cases. In this 
introduction, we will be contended with some properties of M\. Let m% = dim Mi. 

(I) Suppose r,r' > p. Since O(r') contains 0(p) x 0(r' — p), the 0(r' — p) invariants 
r°( r '-P) is an OQo)-module. The stabilizer K\ contains a subgroup isomorphic to 

n — m 

0(p) and Mi restricted to 0(p) is isomorphic to det Q ?-j t°( v ~ p \ In particular 
mi = dime r 0(r '~ p ). 

(II) Suppose r > p > r'. Then i£i contains a subgroup isomorphic to O(r') and Mi 
restricted to O(r') is isomorphic to det ?n r. In particular m 1 = dim c r. 
(Ilia) Suppose r' > p > r, n > m and \x = ' a ^ DL \ p ^ r . Then 9{p) is nonzero and it contains 

n — m 

the i^-type Co( n ) <8> det ?N with multiplicity one. The stabilizer K\ contains a 

n — m 

subgroup isomorphic to 0(r) x 0(n — 2r), and Mi = det ?^ ®Co( n -2r) as an 
0(r) x 0(n — 2r)-module. In particular m\ = 1. 
(IIIb) Suppose r' > p > r, n < m, \x = and r is the trivial representation of O(r'). 
Then #(/i) is nonzero. The stabilizer K\ contains a subgroup isomorphic to 0(r) x 
0(n — 2r). With reference to the notation in Section [2TT| let r' denote the irreducible 
finite dimensional representation of 0(n — 2r) with highest weight ^Y^lp^j.. Then 

Mi = det ?N (8>t' as an 0(r) x 0(n — 2r)-module. In particular mi = dime r'. 

We can now state our main result of this paper. 

Theorem 1.6. Suppose we are in Cases (I), (II), (Ilia) or (IIIb). 

(i) The sheaf A4\oi is the K n ^ r -equivariant locally free sheaf corresponding to the repre- 
sentation Mi of K\. 

(ii) The associated cycle of is mi[Ox] where ni\ = dim Mi. 

The proof is given in Section 17.71 

Part (ii) is an immediate consequence of (i). If r > 2p, then (ii) is Case 1 of Theorem 4.7 
in |NZj . If r > n = 2p and fi = 0, then (ii) is also a result of [T]. For Case (III) in 
general, we have a weaker result, namely Proposition 17.91 which sets an upper bound for 
the multiplicity of the associated cycle. In Section 17.101 we give a counterexample to a 
conjecture of K. Nishiyama. 

Proposition 1.7. In Cases (II) or (III) 

V^ = M = M(0 1 )=hid% r M ei 
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as K njr -modules. 

The proof is given in Section 18.11 The proposition is false for Case (I) because is 
strictly contained in M.{0\). 

In the final section of this paper, we connect our results with Conjecture 12.1 in [V3J 
on unipotent representations. First we review the conjecture. Let x be an element in 
a nilpotent orbit X . Let l x denote the complex Lie algebra of its stabilizer K x . It is a 
complex Lie subalgebra of t. We define a character 7 of K x by j(k) = det(Ad*(fc))|( t /{ ;r ). 
for all k G K x . An algebraic representation (x, V x ) of K x is called admissible if 

(3) x(exp(X))v = 7 (exp(X/2))v 

for all X G t x and v G V x . Vogan conjectured that for every admissible representa- 
tion x of K x , there exists an irreducible unitarizable (q, i^)-module (tt, V n ) such that, as 
a representation of K, 

(4) K = Ind^x- 

This implies that 7r has associated cycle (dim x) [X] . Such modules are candidates for the 
conjectured unipotent ($3, if)-modules attached to X. 
We state a result of Yang |Ynj . 

Theorem 1.8. For (71", V n ) = V$), the character M e = d p 2 in Theorem \1.5\ is an 
admissible representation of K e and, V$ = lnd Ke d p 2 . 

We will supply a proof of the above theorem using our results in Section 16.31 It is the 
desire to extend Yang's result to which provided the initial motivation for this paper. 

Corollary 1.9. We refer to Cases (II) and (Ilia) in Theorem \l.b\ 

(i) In Case (Ilia), the representation M\ is an admissible representation of K\. 

(ii) In Case (II), M\ is an admissible representation if and only z/dimr = 1. 

Furthermore, in the above admissible cases, = Ind^Mi. 

The proof is given in Section 18.51 Finally in Propositions 18.61 and 18. 7\ we identify those 
representations in Corollary 11.91 which are special unipotent representations as defined in 
Chapter 12 in [V2]. 

Most of our methods and results extend to the dual pairs U(p, q) x U(n, m) and 0*(2p) x 
Sp(n, m). We have decided to omit them so as to keep this paper simple. 

Acknowledgment. The first author is supported by an National University of Singapore 
grant R-146-000-131-112. We thank Kyo Nishiyama and Chengbo Zhu for their helpful 
comments. 

2. Local theta lifts 

2.1. Notations. Let N denote the set of non-negative integers. Let l n = (1,1,..., 1) 
and n = (0, 0, . . . , 0) in R n . If A = (Ai, . . . , A n ) G W 1 and f = . . . , f n ) G R m , then we 
denote (Ai, . . . , A n , £1, . . . , £ m ) G R n+m by (A, 0- 
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Irreducible representations of 0(n) are parameterized by arrays of the form 

\i = (ax, . . . , Ofe, O n _fc) or (oi, . . . , l n -2k, Ok) 

in N n where are positive integers, a, > a^+i and < [|]. See [GWJ. We will call these 
arrays weights of O(n). In general, we would ignore the string of zeros in the weights and 
write (a%, . . . , Ofe, n -k) as (ai, . . . , afc) instead. 

Let A(0(n)) denote the set of such weights. Let 7"o( n ) denote the irreducible finite 
dimensional representation of O(n) with highest weight \i. We will denote the trivial 
representation of 0(n) by Co( n )- Finally we recall a branching rule: The dimension of 

Hom ( n )(T"o(n)' r o(n+i)) * s a t m °st one and ^ is one if and only if > /ij > for all 
i — 1, . . . , n. 



2.2. Local theta lifts. First we recall some facts about the local theta lifts of unitary 
characters and unitary lowest weight modules in |Loj . Also see |K0 when p — 1. We will 
assume that n + m is even and (p, n, m) is in the stable range. 

Let W denote the standard representation of Sp(p(ra+m)). We give a complex structure 
on W and realize the oscillator representation on the Fock space Q, i.e. Q = 1Z(W) is 
space of complex polynomials on W . We have the dual pair (G f , G) = (Sp(p, R), 0(n, m)) 
in Sp(p(n + m),R). Let g' and iT'(R) denote the complexified Lie algebra and maximal 

compact subgroup of G' etc... Suppose p' (resp. p) is an irreducible (g', fT'(IR))-module 

(resp. (g, i^(M))-module). The tilde above each group denotes an appropriate double 

cover. Suppose there is a nontrivial quotient map ft — > p' ® p of (g', if'(R)) x (g, if(R))- 
modules. Then a famous theorem of Howe states that the isomorphism class of p is 
uniquely determined by that of p' and conversely. We denote p by 9(p') and call it the 
local theta lift of p' . 

First we set p' = q' to be the one dimensional character of Sp(p(n+m), R) as in Section[T] 
and we consider the local theta lift 6(q'). There are four choices of the genuine character q 
of 0(n, m). Let K n rn = 0(n, C) x 0(m, C). We will fix a choice of q so that $ = q0(q f ) 
has A^-types 

(5) ^=0 r^^mr^ 

i=(h,...,i P ) 

where is sum is taken over I = (li, . . . , l p ) G W such that l\ > 1% > . . . > l p > mm^^ 2 -^ 1 , 0). 
Since we are in the stable range, the module d is unitarizable by |Lilj . The minimal K n m - 

type r min is r 0(n 2 } p ' " p M C ( m ) if m > n and C ( n ) E r 0{ ^ p ' ra p if m < n. 

Let m = r + r'. Since i? is O(n) x 1-admissible, the restriction of $ as a Harish-Chandra 
module of (so(n, r), K n>r ) x O(r') decomposes discretely as a direct sum 



(6) 



O(r')' 
M€A(0(r')) 
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It follows that is unitarizable and it has -fT njr .-types 

(7) m= (r^ lp ' "- p) Kj]m(^,K)ra (r) ) 

l=(h,...,i p ) \ k J 

where the sum is as in (J5j) and m(l, /i, k) is the multiplicity of Kl Tq/.,s in Tq^? -1 ^. In 
particular is nonzero if and only m(l, fi, k) ^ for some I and «. 

Suppose k < p. We lift r o / f.n''' /lfe ' 0r '~ fc to a unitarizable lowest weight module L(fj) of a 

double cover of Sp(p, R). The lowest U(n)-type has highest weight |KVj 

n 

(fxi, . . . , /ifc, Op-k) + — 1 P - 

Let 8(L(t)) be the theta lift of L(n) to 0(n, r). Then 9(L(fi)) is an irreducible and 
unitarizable Harish-Chandra module |Lo] . In addition there exists a genuine character <^ 
of 0(n, r) such that $(//) = $i0(L(t)). 

2.3. Let m = r + r' . One could easily extend the definitions of d and when (p, n, m) 
is outside of the stable range. We will briefly explain their properties and refer the reader 
to |Loj and |LMT] for details. For the ease of explaining, we assume that n < m. We will 
denote ■& by and i%) by $™' r (/i). 

Outside the stable range, $™> m is nonzero if and only if one of the following holds: 

(A) We have n = m < 2p. Then $™' n = i?™!^ which is in the stable range. By (jZJ), if 
CLVpM + 0, then ^(/i) = 

(B) We have p < n < 2p — 1 and m = n + 2. Let 5 denote the one dimensional character 
of 0(n, n + 2) which is deto( n ) on O(n) and trivial on 0(n + 2). Then 5^™'" +2 = fi^-p 2 
and we are back to the stable range. By (j7j), <5i9p' r (/i) = ^"^(/i). 

Finally in Case (A), it is possible that $p' r (/x) 7^ but i?™^ _(//) = 0- Most of these 
lifts $p' r (/x)'s are non-unitarizable. This situation arises because the maximal Howe quo- 
tient 0(V) is reducible. It is possible to analyze of the i^-types of 0(V) as in [LoJ and 
compute its associated cycles. This is tedious so we will omit this case. 

3. Locally free sheaves 

In this section, we will state a useful proposition on local free sheaves over co-adjoint 
orbits. All Lie algebras are complex Lie algebras and all Lie groups are complex linear 
groups. Let q = so(n + m, C) and let g = 6 © p denote the complexified Cartan decompo- 
sition. Let S(p) = Sym(p) be the symmetric algebra. It is also the algebra of polynomials 
K(p*) on p*. Let K = K n , m = 0(n, C) x 0(m, C). 

Let X be a co-adjoint i^-orbit in p*. We pick an x G X. Let K x denote its stabilizer 
in K. Let q : K — > X defined by q(k) = (Ad*A;)x. Let W be a finite dimensional C- 
linear rational representation of the stabilizer K x . For an open subset U of X, let n^U) 
denote the algebra of regular functions on The group K x acts on H(q~ x (U)) ®c W 

in the following way: Given s G K x , f G IZi^q^iU)) and w G W, we set f s (k) = f(ks) for 
all k G q~\U). Then s maps / ®wto/ s ® sw. We set £(U) = (K(q- l {lJ)) ® W) K * to 
be the subspace of ^-invariants in TZ(K) £g> W. This defines a locally free K-equivariant 
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coherent sheaf £ = Cx.w on X — K/K x . See Sections 1 and 2 in [CPS] for details. In 
particular we define hid%W = C(X) = (TZ(K) © W) K * and it is a left ft (A*) -module. 

Let Z = X denote the Zariski closure of X . It is an affine complex variety in p*. We 
consider B = C(X) as an ft(Z)-module and we let B be the corresponding coherent sheaf 
on Z. Let r x : B — > W denote the composite of the maps B = C(X) — > C x — > C x /m x C x = 
W where m x is the maximal ideal at x. 

Proposition 3.1. We assume the above notation. Suppose r x is surjective. 

(i) Then C x = 1Z X ®n{z) B = B x . Furthermore B\x — C as sheaves on X . 

(ii) For any f G TZ(Z), let Uf be the corresponding principal affine open subset of Z. 
Then B{U f ) = B f = £{U f n X). 

(Hi) We have B{U) = C(U fl X) for any open subset U C Z . 

Proof. Since r x is surjective, 1Z X <S> C(X) = C x by Nakayama lemma. This proves the 
first assertion of (i). The second assertion of (i) holds because both B\x and C are K- 
equivariant sheaves with the same stalk at x e X. This proves (i). 

By (i), if Uf C X then (ii) holds. Now we prove the general case. First we write 
X = [Jl =1 Uf H as a union of open affine subsets where hi G TZ(Z). Then we have the 
following exact sequence 

o _> B -> 0£(%J Q)C{u hi n u hj ). 

i i<j 

We will localize the above with respect to /. Now Ui H and U^f = U^ fl Uf are affine 
opens subsets in X. Hence (ii) applies and we have 

C(U hi ) f = (B hi ) f = B hif = C(U hif ) = C(U hi n Uf). 

Similarly we get £(£7/^ fl Uf^)f = £(£7^^/) = £(L^. fl Uh 3 H Uf). Since localization is an 
exact functor, we get an exact sequence 

-* B f -> ^ n u f) n u hj n i/,). 

This proves (ii) because Ui=i(^< ^ ^/) = X HUf. Finally 

B(17) = \jm UfCU B(U f ) = \jm UfCU C(U f n X) = C(U n AT). 

This proves (iii) and the proposition. □ 

3.2. Remark. In the above proposition r x would be surjective if C(X) ^ and W is 
an irreducible representation of K x (for example when dim^ W = 1). Indeed we pick a 
nonzero (p G C(X). Then by the i^-action, we may assume that tp(x) ^ in VK. Hence 
r x 7^ 0. The map r x is surjective because W is an irreducible representation of K x . 

4. Good filtrations 

In this section we will construct good filtrations using local theta correspondences. 
Unless otherwise stated, all Lie algebras and Lie groups are complex Lie algebras and 
Lie groups respectively. We set g = 50 (n + m, C) = t © p be the complexified Cartan 
decomposition, K = K Ujm , g' = sp(p, C) = ¥ © p' and K' = GL(p). We will identify p* 
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with the space of n by m complex matrices M n m . A tilde above a group will denote an 
appropriate double cover which is usually clear from the content. 

First we review Section 3.3 in Nishiyama and Zhu |NZj . 

We recall the oscillator representation of Sp(p(n + m)) on the Fock space Q = 1Z(W) 
of complex polynomials in Section 12.21 We follow Howe's notation [H2J about diamond 
dual pairs. Let (M(R),K'(R)) = (U(n,m),U(p)) be the dual pair in Sp(p(n + m),R). 
Let %m © (nx'- 2 ' -' ©m'- ' 2 -') denote the Cartan decomposition of the complexified Lie algebra 
of M(R) such that m^ 2 ' ^ acts by multiplication of K' -invariant quadratic polynomials on 
W and m^ ' 2 ) acts by degree two fT'-invariant differential operators. 

Fact 3 in Howe's paper [H2J states that in sp(p(n + m)), we have 

(8) m( 2 <°)©m(°' 2 ) =p©m(°' 2 \ 

The projection of p to m^ 2 ' ^ under the decomposition of the left hand side of (JHJ) is a 
/^-isomorphism. We identify p as m^ 2 ' ^ = (M n m )* via this projection. 

We also have the dual pair (M'(R),K(R)) = (Sp(p,R) x Sp(p, R), 0(n, R) x 0(m,R)) 
in Sp(p(n + m),R). In a similar fashion, we have a Cartan decomposition 

/ h _ / /(2,0) _ /(0,2)n , / ^ /(2,0) 

m = tM> © (m © m v ; ) and p = m v ; 

where p' is the non-compact part of q'. 

The Fock space Q has a natural grading Q d = {/ e 7?.(Vr)| deg/ = c?} and a natural 
filtration = ©J =0 fK Let Grfi = 0°l o %M-i be the graded module. Then m (0 ' 2) 
acts by zero on Grfi. By (JHJ) the p-action on Grf2 is equivalent to the m'- 2,0 -' -action. 
Similarly the p'-action on Grf2 is equivalent to that of m'^ 2 '°\ 

Suppose p' is an irreducible i^')-module which occurs in the theta correspondence. 
In other words, there is a quotient map rj : Q — > p' <g> 9 where 9 = 9(p') is an irreducible 
(q, i^)-module called the local theta lift of p'. Let jo be the first j such that r](flj) ^ 0. 
By |H2j . p' ® 9 is generated as a U(q') x £/(g)-module by the image of a joint harmonics 
a' '®cr in degree jo- Here a' and a are irreducible representations of K' and K respectively. 
We will call a 1 the if'-type of lowest degree for p' . We take a dual vector v'* corresponding 
to a vector in the lowest degree a' isotypic component of p' . Let v '* © id : a' © 9 — > 9 be 
given by v' © w i-> (?/*, and let 

(9) z/: 12 — )■ p ®9 — > a ®9 — > 9 

denote the composite map. 

We define Fj = Gj = Uj(g)u(Q jo ) and Gr 9 = 0^L O Gj/Gj-i- ^ i s c l ear that Gr 9 

is an (S(p), K)-module, and Gj is a good filtration of 9, i.e. Gj+% = gGj and UjGj = 9. 

Lemma 4.1. We have F 2 j + j 0+ i = F 2 j + j and F 2 j + j = Gj. 

Proof. The map v factors through the i^'-covariant subspace Q a > of type a' of Q. Let 
fl(o~'), Q d (a') and Qj(a') = ©d<j^ d (c") denote the a'-isotypic components of Q, Q d and 
Qj respectively. Since K' has reductive action on Q and preserves degree, fi(cx') maps 
bijectively onto the covariant Q a i. Moreover Q d (a') = Q d r\Q(a') and Qj(o~') = Qj C\Q(a'). 
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Hence 

(10) u(n j (a')) = F r 

Let %(a') denote the cx'-isotypic component in the harmonic subspace H(K') of 1Z(W) 
for K'. By [H2J, we have U{a') C Q jo and by [HI] we have 

(11) Q(a') = Sym(m (2 ' 0) )H(a / ). 

We observe that m^ 2 '^ is abelian so Uj(m^) = J2 d=0 Sym d (ra ( - 2 ' ^). Since m( 2 ' 0) are 
degree two polynomials, fP (cr') = if j ' ^ jo (mod 2). It follows from (fTu| and (fTTj) that 
v(U 1 ( m ^)n(a')) = F 2j+JO = F 2j+JO+1 . 

We will prove Gj = F 2 j+j by induction. First we have G = Fj . Suppose Gj-i = 
F 2 y_i) +J - = i/(fi') where Q' = n 2 (j-i)+j - Since Gj C F 2 j + j , if suffices to show that 
*W CGj. By (gj), fl' + m^ 2 ' ^' = O' + pO'. Hence 

F 2j+j0 = */([/, (m (2 ' 0) )H (a')) C + m (2 ' 0) fi') = + ptf) = v(Qf) + pz/(fi') 
= Gj^i + pGj-i = Gj. 
This shows that F 2 j + j = Gj and completes the proof of the lemma. □ 

Remark. The first equality of the above lemma was proved in [NZj . The second equality 
was assumed without proof in that paper. 

Let Gr d 6 = F d /F d ^ and Gr6 = ® d Gr d 6. It is a graded (S(p), ^)-module if we 
consider p has degree 2 and Gr d 9 is non-zero only if d = 2j + jo for j > 0. This grading 
is essentially same as the one discussed in Section 11.11 and in |Vlj modulo the shifting in 
degrees. We will use this new grading for the rest of this paper. 

Let Mp tm denote the space of complex p by m matrices. Let W + = M p>n , W~ = M PiTn 
and W = M Ptn+m = W + x W~ . We define the K x i^'-equivariant quotient maps JNZj : 

4>: W = W + x W~ -> M n>m (C), ip: W = W + x W~ ->■ M PiP (C) x M PiP (C) 

{A,B)^A T B (A,B) ^ (AA T ,BB T ). 

Now m^ 2 ' ^ can be identified as the dual of M n>m . For any h E m^ 2 '°\ h acts as multi- 
plication by the degree two polynomial h o <\> on 1Z(W). Similarly h! G (m')( 2 '°) acts as 
multiplication by degree two polynomial h! o ip on T^W). 

5. The null cones and orbits 

In this section, we study some relevant nilpotent orbits. All groups in this section are 
complex linear groups. 

Let M n>m denote the space of n by m complex matrices. Let Eij denote the matrix 
whose all entries are zero except its (i, j)-th entry is 1. Let C n be the quadratic space 
with bilinear form x\ + . . . + x 2 . Let {fx, ... , f n } denote the standard basis of C n . Let 
O(n) = 0(n, C) be the corresponding orthogonal group defined by the quadratic form. 
Similarly we have C m with the O(m) action. Let {e 1; ...,e m } be the standard basis 
of C m . Let C r denote the quadratic subspace spanned by {e 1; . . . , e r } and let O(r) denote 
the corresponding orthogonal group. 
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For 2p < n, we define a p by n matrix 

E(p,n) := (E u + . . . + E pp ) + y/^l(E 1 

,n— p+1 

+ ... + E p 



(12) 



/I 



lo 







n 







\ 



v 



(vi, • • • , v p ) is the n by 



For j = 1, . . . ,p, we set Vj = f, + v^Tfn-p+j- Then _E(p, n) 
p matrix with the v/s as column vectors. 

Let i? + = E(p,n) T above and let M+ m := E + M PtTn in M„ jm . The column space of 
M+ m is the isotropic subspace V of dimension p spanned by the v/s. Let V* denote the 

column space of E + M p ^ m . Let P p denote the stabilizer of V . Then it is a maximal parabolic 
subgroup with Levi decomposition P p = L p ix N p where L p = GL(V) x 0(n — 2p) is the 
Levi subgroup which stabilizes V and V*. We identify GL(p) ~ GL(V) in the following 
way: For g G GL(p) we associate an o g £ GL(V) C O(n) by formula 



(13) 



o 9 E- 



E + g. 



The action of P p on M+ m induces an action of P p on M Pt7n , namely, 0(n — 2p) and iV p 
acts trivially on M Pi?n and GL(p) acts by left multiplication. Likewise, by replacing n by 
m, we define P' p = L m k N' p in O(m). 

Let I © p be the Cartan decomposition of the complexified Lie algebra of the real 
Lie group 0(n, r). We will identify the complex space p* with C n <g) C r = M njI .. The 
complex linear group K n>r = O(n) x O(r) acts on M„ jr via the co-adjoint action by 
Ad*((?i, g 2 )(X) = giXg^ 1 where g 1 G O(n) and g 2 G O(r). Similarly GL(p) x O(r) acts 
on M P)T by Ad*(^i, g-i){X) = g\Xg 2 l where G GL(p) and ^ 2 e O(r). 

5.1. Notation. We will be using the following notation in the next few sections: 

Let (p,n,m) be in stable range and let m = r + r'. We set E + = (E(p,n)) T and 
E = E(p, to) as above. Let e = E + E. For any < g < min(p, r), we define 

(14) Af(p,r;q) = jlG M Pi ,, | rank(X) = min(p, r), rank(XX T ) = q } and 
0(p,r;q) = { oE + X G M n , r | o G O(n), X G A/"G»,r;g) } . 

Note that 0(p, r; g) = (AdK ntTn ) E + N{p, r; g). 

Let Af = Af(p, to; 0) and O = 0(p, to; 0). The Zariski closure AT is called the null cone 
in M PiJn . If q = min(p, r, r'), then we denote Afi = Af(p, r; g) and Oi = 0(p, r; g). Let A/" 
denote the GL(p) x O(r) x 0(r')-orbit of E in M p>m . Let O denote the 0(p) x O(r) x O(r')- 
orbit of e in M n _ m . 

Let pr : M p 



p,m- 



m — >■ M p f , denote the projection map by deleting the last r' columns of 
Let Ei = pr(E). Similarly let pr^ : M n>m — > M n r be the projection map defined 



by deleting the last r' columns of M n>m . Let e\ = E + E\. 
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Lemma 5.2. The subset M\ is a single GL(p) x 0{r)-orbit generated by E\ and 0% is a 
single K nr -orbit generated by e\. 

The orbits O and 0\ agree with the ones defined in Proposition 11.21 and in Sec- 
tion [L3] respectively using signed Young diagrams. Clearly pr K e = e±, pr^(C ) = 0\ and 
pr(Ao) = M\. We will show later in Proposition 15.51 that pr x ((9) C O x and pr(AT) C ~M\. 
The following two diagrams are instructive: 

M n , m D O D C 3 e M Pjm D TV D W 9 £ 

P r A' 

M n , r D Ci D 0i 9 ei M Pjr D Mi D M 9 

Proof of Lemma I5.M The assertion about d follows immediate from the first assertion 
on Mi- For Mi, Proposition 3.1 in |Knj proves the case p < r while Proposition 3.7.2 in 
|H3j proves the case p > r. We give the proof for completeness. First we recall Witt's 
theorem in Section 3.7.1 in [H3J: 

Theorem 5.3 (Witt's Theorem). Let T\,Ti G M r ^ p be two complex matrices. Then there 
exists an o G O(r) such that oT\ = T2 if and only if 

(i) T^T\ = T 2 T T 2 and, 

(ii) kerTi = kerT 2 where we consider Tj : C p — > <C r as a linear transformation. □ 

We begin the proof of Lemma 15.21 It is clear that E\ G M\. Let X G M\. There is 
a g G GL(p) such that gXX T g T = E\Ej because GL(jo)-orbits on the space of p by p 
symmetric matrices are parametrized by rank. We may replace gX by X so that XX T = 
E\Ej . We will show that X T and Ej satisfy Conditions (i) and (ii) in Theorem 15.31 The 
lemma will follow from the theorem. 

Condition (i) is trivial. Now we check (ii). First suppose r > p. By definition X T has 
rank p. Hence kerX T = ker Ej = 0. Next suppose r < p. Then p < r' . Hence X T and 
XX T have the same rank by definition. By dimension counting kerX T = kerXX T = 
ker EiE J = kexEj. This gives (ii) and completes the proof of Lemma 15.21 □ 

Given a Zariski open subset Z of an affine variety, we will denote its ring of regular 
functions by 1Z(Z). Suppose Z is a cone in M Ptm with the natural C x action by given by 
scalar multiplication. Then we set 

n\z) = { f g n{z) I f(cX) = c d f{x) Vcgc.igz} 

to be the degree d regular functions. Thus we have 

n{z) = ^n\z). 

den 

For example, if Z = M pm , then TZ d (M p>m ) is the usual degree d polynomials on M pm . 
Suppose G acts on the cone Z and commutes with the C x action, then lZ d (Z) is a G- 
module for every d G Z. We can now state some results in Section 3.7 in |H3j . 



pr 
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Proposition 5.4. (i) Suppose m > 2p. Then as a GL(p) x 0(m) -module 

^ ( A7) = 0( r A L(p) )* 0r A (m) 
A 

where the sum is taken over A = (Ai, . . . , X p ) G N p such that Aj > Aj + i and Yli=i ^ = 
d. ' ^ 

(ii) Let I be the ideal in lZ(M p>m ) generated by degree two O(m) -invariant polynomials. 
Ifm> 2p, then 11(77) = K(M p>m )/I. 

(m) Ifm> 2p, then K(JSf) = 11(1/). □ 

Proposition 5.5. (%) We have Ao = (pr _1 A/i) fl A" and it is a Zariski open and dense 
subset in A". In particular, Ao = A/". 

(ii) The Zariski closure of the image pr(A") is AT- 

(raj M^e /iai>e (9 = (pr^Oi)) H and it is a Zariski open and dense subset in O . In 
particular, Oq = O. 

(iv) The Zariski closure of the image pr K ((D) is 0\. 

Proof. (i) By the definition of A/i and Lemma 15.21 we have pr Ao = A/i . Hence Ao f= 
(pr _1 (A/i)) n A". On the other hand, let X G pr^Ai) PI A". We set X x and X 2 to be the 
first r columns and the last r' columns of X. Since X G A/", 

(15) X x Xj + X 2 Xj = 0. 

By the action of GL(p) x O(r), we may assume X\ = E\. Let E 2 be the last r' columns 
of E. By (USD, X 2 Xj = -X x Xj = -E x Ej = E 2 Ej . We claim that there is an o G O(r') 
such that oXj = Ej. For this, it suffices to verify Theorem I5.3( ii): 

1. If p < r, r', then kerX^ = = ker Ej since ker X 2 Xj = 0. 

2. If r' < p, then kerX^ = ker Ej by the same argument for the second case of 
Lemma 15.21 

3. If r < p, then kerX^ C ker X 2 Xj = C p ~ r . Since X T has rank p, the last p — r 
columns of X T are linearly independent. However the last p — r columns of Xj are 
zero. Hence the last p — r columns of Xj are linearly independent. This means 
kerX 2 T = = ker Ej . 

This proves our claim. The claim implies that X G Ao- This proves Ao = pr _1 (Ai) C\J\f. 

By above argument, Ao is the subset in M such that X\ has full rank and XiXj 
has maximal possible rank. Since rank condition is given by the non-vanishing of the 
determinate of minors, Ao is an open dense subset in Af. Moreover Ao is open and dense 
in A" since A" is open and dense in A. This finishes the proof of (i). 

(ii) By (i), we have pr(A') C AT- Then (ii) follows because both subsets contain the 
GL(p) x 0(r)-orbit Af\ generated by E\. 

(iii) It is clear that O C (pr^-^Oi)) n O. Given Y G (pr^-^Ci)) n O, we may assume 
that Y = E + X for some X G M by the 0(n)-action. Now Y x = pi K (E + X) = E + X x = 
oE + X[ G 0i for some o G 0(n) and X[ G Ai.. We claim Xi G Ai. Indeed \(E+) T E + is 
the identity p by p matrix. Therefore rankXi > rank (Y\) > rank (|(£ ,+ ) T Yi) = rankXi. 
Applying same argument on X[, we get rankXi = rankYi = rankX(. Similarly, we have 
mnkXiXj = rankY^ 1 " = rankX(A( T . Therefore X x G Ai by definition $H]). Now 
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Oq = (pr^- 1 (0i)) PI O follows from (i). The rest of (iii) follows by the same argument as 
that of (i). 

(iv) It is similar to that of (ii). 

□ 

5.6. Stabilizers. We define the following stabilizers: 

(1) The stabilizer S of E G JV in GL(p) x O(m). 

(2) The stabilizer S of E G Af in GL(p) x O(r) x O(r'). 

(3) The stabilizer Si of E 1 G M in GL(p) x O(r). 

(4) The stabilizer K e of e = P+P G £> in O(n) x O(m). 

(5) The stabilizer P of e = E + E G C in O(n) x O(r) x O(r'). 

(6) The stabilizer K x of e x = P + P x G d in O(n) x O(r). 

Clearly S = Sn (GL(p) x O(r) x O(r')) and K = K e n (O(n) x O(r) x O(r')). It is also 
easy to see that So C Si x O(r') and if C i^i x O(r'). 

The stabilizers S, Sg and We recall that = (GL(V) x 0(m - 2p)) x iV p is the 
stabilizer of the column space of E T in O(m). We recall (fT3]) where g G GL(p) corresponds 
to o g G Pp such that o 9 P T = E T g. Now it is easy to see that 

(16) S= { (g.o^-iyhu) G GL(p) x ?; | 9 e GL(p),/i G 0(m-2p),u G iV p } . 

Lemma 5.7. VFe /jave 

Si = { (9i,92) G GL(p) x O(r) | (^^ 2 ^ 3 ) G S c GL(p) x O(r) x O(r') }. 

Clearly Si contains the right hand side. The proof that S\ is contained in the right 
hand side follows the same argument as in the proof of Lemma 15.21 We leave the details 
to the reader. 

We will describe So and Si explicitly later in Section 17.31 

The stabilizers K e , K and K\. If (gi,g 2 ) G K e C 0(n) x O(m), then g\ stabilizes the 
column space of e which is the span of {vi, . . . , v p }. Hence g\ lies in the maximal parabolic 
subgroup P p = (GL(V) xO(n — 2p)) xN p . In a similar fashion g 2 G Pp. Together with ([IB"]) , 
we get (JTJ and 

P e = {(o g xu,g 2 ) G P p x 0(m)\(g,g 2 ) G S, x G 0(n - 2p), u G iV p } 
( ^ =(5xO(n- 2p)) x jV„. 

Since S = 5 n (GL(p) x O(r) x O(r')) and K = K e n (O(n) x O(r) x O(r')), we have 
P = (5-q x 0(n - 2p))JV p . 

Let P( = (Si x 0(n — 2p))N p . If p < r, then a similar proof as that of (1171) gives 

K 1 = K[ = { ( 9l ,g 2 ) G O(n) x O(r) | (<?!, <? 2 , £3) G P C O(n) x O(r) x O(r') }. 

Suppose p > r. Let P r = (GL(r) x 0(n — 2r)) x iV r be the maximal parabolic subgroup 
in O(n) stabilizing the span of {fi, . . . , f r }. Then 

Ki = {(o g xu, g) G P r x O(r) : g G GL(r), x G 0(n - 2r), u G iV,.} ~ P r 

and it contains K[ properly. 
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6. Associated cycle of $ 

The objectives of this section are to prove Theorem 11.51 and Theorem 11.81 

Let K = K n ra = 0(n) x O(m). We recall the one dimensional genuine characters <; 

and in Sections [1] and |2] such that $ = ^(Y). We have the quotient map v = r\ : 
— > <?' <8> 6 of (sp(p), U(p)) x (so(n, m), A)-modules as in ([9]). We prefer to work with 

K- modules instead of A-modules. By tensoring with := W)" 1 ® *>■> we have a quotient 

map 

(18) 77o = <;oV ■ 9>fl 1 ® 

which is a homomorphism of (sp(p), GL(p)) x (so(n, m), i^)-modules. We realized the 
Harish-Chandra module $ on the same vector space as that of #(<?')• The good filtration 
Gj = F2j+j on the space of #(?') defined at the end of Section H] induces a good filtration 
on We will continue to denote this filtration on i9 by Gj = F 2 j + j . One checks that the 
joint harmonics form the minimal K-type r min of ■&. Hence Gj = F 2j+jo = Uj( 

fl) ' Train- 
Let M = Gr$ = ®dFd/ F d _i denote the graded module. It is an (S(p), A)-module. 

We continue to realize s Q VL on 1Z(W). We describe the actions of K' = GL(p) and K n>m 
actions on it explicitly: For (g,gi,g 2 ) G GL(p) x O(n) x O(m) and / G 72.(W), 

{(9,91,92) ■ f){zx,z 2 ) = det^ (g)f(g T z 1 g 1 , g~ l z 2 g 2 ), V(zi.z 2 ) G W + x W~. 

We recall p' = TZ 2 {W + )° {n) © ft 2 (W~)° (m ). We consider the following diagram: 

X__^ 

Ux®U-£ *K(W) =K(W + )®K(W-)— ^M = Grtf 

^--^^^ Grr/o ■■■ ^"^ 

^(A7+) ® TZ(J\F) - >(ft(A7+) ® K{N~)) K ' 

where I = p'7?.(H / ) and A/" 1 * 1 are the open null cones in W ± . Note that J\f~ = M in 
Section 15.11 

We have assumed in Section [T] that (p,n,m) is in stable range. By Corollary 3.7.3.6 
in |H3j , X is precisely the ideal of polynomials vanishing on the null cone Af + x J\f~. This 
shows that middle- vertical line in the above commutative diagram is exact. On the other 
hand, p' C q' acts trivially on 1?, so X has zero image in M. Hence the map Gr 770 in the 
above diagram is well-defined. We recall : W — > M ntTn (C) defined at the end of Section H] 
and O = 0(p, m; 0) = <fi(N + x A/" - ). Hence if / G S(p) vanishes on O, then / o <p vanishes 
on A" + x J\f~ so / o (p g X. From the commutative diagram above, / acts trivially on M. 
Hence the S(p) action on M factors through TZ(0), and M is an 7£(C?)-module. 

Proposition 6.1. The surjection^ : (TZ(J\f + ) ®lZ(J\f~)) — > Gvd is an isomorphism of 
(S(p), K n>m ) -modules. 
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Proof. Since K' acts trivially on Gr$, Grr/o factors through the K' co- invariants of 
1Z(N + )®1Z(N~). On the other hand, the K' action is reductive and the if-invariant sub- 
space surjects onto the co- invariants. By Proposition I5.4H ). we have following f^-module 
isomorphism 



(19) = ^ (B) ®7^ (m) 

where the sums are taken over \i = (//i, . . . , fj, p ) and A = (Ai, . . . , X p ) in N p such that 
A*i > ■ ■ ■ > > and Ai > • • ■ > A p > 0, 

The .fT-types of the domain and the image of £ are the same as those of by com- 
paring 05]) and ( 1X9]) . Therefore we conclude that the surjection £ is an isomorphism. 
The symmetric algebra S(p) acts as multiplying /^'-invariant polynomials, so £ is also an 
S(p)-module homomorphism. □ 

We note that the summand Tq,^ ®To(m) m occurs m degree Ei/^j + E« Hence 
when m > n, 



\ I m — n -I 

(20) Gr^ = 0r^ B) » lp ® r* (m) 

A 

where the sum is taken over A = (Ai, . . . , X p ) E W such that Ai > • • • > \ p > and 
d = 2(J^| =1 Aj) + pi^Y 1 )- We have a similar formula when n > m. 

Let £ be the if-equivariant invertible sheaf on ~ K/K e defined by the one dimen- 

n — m 

sional character d p 2 of K e in The space of global sections on O is (see Section [3]) 

£((9) = Ind*(d^). 
It is an (1Z(0), if)-module and hence an (S(p), if)-module. 

Proposition 6.2. The graded module M and C(0) are isomorphic (S(p), K nym ) -modules. 

Proof. We set k = ((71,(72) E O(n) x O(m) = if throughout this proof. We construct an 
embedding of (1Z(0), if)-module as follows: 

jy-/ n — m 

(nQ7+)®n(KF)) 4 ind£ e d p 2 

Ei /i ® ^ •> (/•:/••> E j ./•((/•:•)■(/,!//,(/•:.(/,)) . 

For s = (ogXiUi, or g -i\TX2U2) E K e as in (JT]), 

F(fcs) = ^2f j ((g 1 o g xiu 1 E + ) T )h j ((g 2 o {g -i ) TX2U 2 E T ) T ) 

j 

= Y,^9 T {E + ) T g V 1 )h j {g^Eg r 2 ) = J2^^(g- 1 )f J ((E + y g J)h 3 (EgJ) 

j j 

n — m 

= d p 2 {s~ l )F{k). 
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Hence F is in the space on the right hand side, and i is well defined. 

Since ((E + ) T ,E) generates the dense K orbit Af + X Af~ in Af + x AT", the map i is 
injective. 

We check that the above map is an S(p)-homomorphism. Indeed 
i(h -f)(k) = ho T gJ, EgJ)f((E+) T gJ, EgJ) = h( 9l E + EgJ)f((E + ) T gj , EgJ) 



(h-L(f))(k) 



where / G (7^(AT+) g) TZ(J\F)) , h E S(p), fc = (#1, g- 2 ) € It is easy to check that the 
S(p)-action is compatible with the inaction. 
We compute the if-module structure of C(0): 

C(Q) = bd°W«0W M a^,<n*) det ^ 

\A=(A 1 ,-,A p )eNP 
= (37 T 0(n) 2 P ® r O(m)- 

Since t is an injection, it is an isomorphism by comparing the above with ffTT?]) . □ 



We will now prove Theorem 11.51 Let M. be the coherent sheaf of M on O. The last 
proposition proves Theorem ll.5( iii). By applying Proposition I3.1( i) and Remark 13.21 to 
the last proposition, we have Theorem II. 5( i). Part (ii) follows from (i) and the definition 
of associated cycles. This proves Theorem 11.51 

6.3. We give a proof of Theorem 11.81 using Theorem 11.51 Let x = e = E + E e O. Then 

(t/(Q)* = c p ® c n - 2p © a 2 c p © (c p y © c m ~ 2p © a 2 (c p )* © ( B i{p) © qI( p )/AqI(p)Y 

as a representation of AGL(p) x 0(n — 2p) x 0(m — 2p) in the stabilizer K e . Hence 

7 = det £™ © det^ (n _ 2p) © det£ (m _ 2p) 

n — m 

where the unipotent part of acts trivially. If we set \ — dp 2 as in (J2J), then (j^J) holds 

n—rn 

and dp 2 is an admissible representation of K e . The rest of the theorem follows from 
Theorem 11.51 This proves Theorem 11.81 

7. Associated cycle of 
The objective of this section is to give a proof of Theorem 11.61 

We recall 7] : %Q -> 1 © i? and c fi = ft(W) in (HSJ. We will denote r£ (r/) by r. Let 
/ : r — 7- C denote a nonzero linear functional. Let pr : d — > $(//) ©t denote the projection 
to the summand in dHJ). Let z/ : <Jo^ — > denote the composition v' = (id © /) o pr. 
We set Fj = u'^flj) = pr(Fj) where F 3 - is the filtration of i? defined after ( TT8|) . Let jo 
be the minimal j such that F'- 7^ and let G' Q := Fj Q . If n < m, then by (12"0|) . it is not 
difficult to see that G is the minimal i^ nir -type of However if n > m, G' may not 

necessarily be the minimal l^y-type. 
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Let 0i = so(n + r, C). For j 6 N, we define the subspace G' = Uj(gi)Fj Q = Uj(gi)G' 
of Both Fj and G'j are nitrations of 



Lemma 7.1. H^e /love -F 2 'j+i+jo 



r 2j+j 



Proof. The first equality follows from Lemma 14. 11 Clearly G" 7 C i^j+jo ■ ^ remains to 



show that G'j D F^ j+jQ . 



The genuine characters q, q±, qo etc... in (ITS1) does not affect the lemma. Hence we 
will ignore them in this proof. The reader could easily insert them when necessary. For 
example, we will denote q Q by Q and 9(L(fi)) by $(//). 



We decompose W = W x W" where W 



M Pjn+r and W" 



M p r i are spaces of 



complex matrices. We set fi' = and fi" = 7£(W")- Then ft = = O' <g> fi". 

We recall Q d and in Section H We define (ft') d = ^ d n ft' and fij 
natural grading and filtration on Q' respectively. Similarly we define (Q 
We have 



n D,' to be the 
and fij in fi". 



(21) 



fi d = (0') a g> (O 



w\6 



a+6=d 



The Lie algebra $j' = sp(p, C) acts on Q' and fi". 

We decompose Q" = ® , L{ji')* ® r M ' with respect to the dual pair (Sp(p, R), O(r')) 
where the sum is taken // run over all irreducible finite dimensional representations 
of O(r'). Let L = L(n). Let h : fi' — >■ ® L be the usual Howe correspondence 
map. We consider the following diagram: 



f/0 



0- 



. fi' ® (L(//)* 



r; 



0(r'V 



ft' g> (L* ® r) 



h®id 



1%) ®L®U 



i(K/ 



r — »- #(//) Cg) r 



The map n : L g) L* — >■ C is a jj'-invariant pairing. First v' = pr o 770 factors through the 
//-isotypic component of Q, i.e. the dotted diagonal arrow exists. Since g' acts trivially 
on 0, v' will factor through ® r, i.e. the vertical dotted arrow exists. 

Let a' be the K'-type of lowest degree for L(/i). Let j' be the degree of a' in fi' and 
let j'q be the degree of r in f2". With reference to f[2"Tj) . j — Jo + Jo • 



We note that (fl 



Jo- 



(a')* <S> /i. Using Lemma KT\ v'{£l' Q 



Let e = 0,1. By pi], pr^,, 



a/2]" 



g -invariant, 



pr u(Uj" Since the maps 770 and 1/ are 



-.A) 



C 



u h tf)n iS ) = v'(u b ( e ')n' a+i , ® %) 



■J6- 



3o- 



G( 



a/2] +6- 
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Hence F 2 j + 2 = z/(O 2 j+j ) C G'j as required. □ 

In order not to confuse with the graded module M = Gr $ in the Section [HI we will 
denote the graded module Gri%) = ©~ (Fj/Fj^) = ®°° =0 (l o pr(F i /F i _ 1 )) by ikP. 
Then 

(22) M M = Hom ( r ')(r, M). 

Let M n>m be the space of complex n by m matrices. Since S(M n>r ) commutes with O(r') 
in S(M„ >m ), ([22]) is an isomorphism of (S(M„ ir ), _K"„ >r .)-modules. By Proposition I5.5l (iv). 

pr x (0) = ~Q[ and pr^ : K{p~ x ) K{0) = TZ{Oq) is an injection. The S(M„ >r )-action 
factors through 1Z{0\) and (122]) is an isomorphism of graded (7Z(Oi), fT n)r )-modules. 



7.2. . Proof of Proposition \l-4\ Let Zi be the associated variety of $(//). Since M M is an 



7^.(01 )-module so Z\ is contained in 0\. On the other hand is a submodule of By 

Theorem 3.1 in [Ko] , Zi contains the closure pr K (0) which is 0\ by Proposition I5.5l( iv). 
This shows that Z\ = 0\ and proves Proposition 11.41 □ 



7.3. The stabilizers S , Si and T. We recall Section Ell that S C Si x O(r'). Given 
r = To( r /\, we define a representation of T of Si by 

(23) T:=Homo( r .)(r,Ind| xO(r,) C 50 ). 

This representation will enter our main theorem. 

We will compute So = S H (GL(p) x O(r) x O(r')) and the representation T explicitly. 
We divide into Cases (I) to (III) according to Section 11.31 The proofs are not difficult so 
we will omit most details. We write 

(24) g = ( 9l ,g 2 ,g 3 ) E GL(p) x O(r) x O(r'). 

Case (I): r,r' > p. Let 0(p) x 0{r — p) denote the subgroup of O(r) stabilizing the spans 
of {e 1; . . . , e p } and {e p+1 , . . . , e r }. Similarly let 0{p) x 0(r' — p) denote the subgroup 
of O(r') stabilizing the spans of {e m _ p+1 , . . . , e m } and {e r+1 , . . . , e m _ p }. Then S consists 
of all g = (gi, g x g 2 , gig 3 ) as in Q2U) where g x E 0(p) in GL(p), #i# 2 e 0(p) x 0(r -p) 
in O(r) and G 0(p) x 0(r' — p) in O(r'). By Lemma [5.71 Si = AO(p) x 0(r — p). 
Then T = 

r O(r'-p) as a 

AO(p)-module, and 0(r — p) acts trivially. 

Case (II): r > p > r'. Let Q = (GL(p — r') x GL(r')) x iV' denote the maximal parabolic 
subgroup of GL(p) stabilizing the span of the bottom r' rows of E, i.e. the span of 
{f p _ r /_i_i, . . . , f p }. We will denote a matrix in GL(r) with respect to this basis as 

w ft: 

where &n is a p — r' square matrix, 6 2 2 is a r' square matrix. 

We perform a change of basis for C r . Let e'- = e 3 - + -\/— l~ e m- P +j and e^__ p+J - = — 
v 7 — Te m _ p+ j for j = 1, . . . ,p — r' . Let e!- = ej for j — r' + 1, . . . , m — p. We write g E O(r) 
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with respect to this new basis as 

' (111 dyi Ql3 a ll\ 

^2g^ _ °21 a 22 a 23 a 24 

9 O31 032 CI33 CI34 

ya4i 042 CI43 044 J 

where an is a p — r' square matrix, a 2 2 is a r' square matrix, a 33 is a m — 2p square and 
044 is a p — r' square matrix. Let 

/ h 7i 

fc= J 2 

where I\ is the p — r' by p — r' identity matrix and I 2 is the r + 2r' — 2p by r + 2r' — 2p 
identity matrix. The matrix of g with respect to the standard basis is given by hX g h~ l . 

Lemma 7.4. The stabilizer Sq consists of all g = (<7i, #2, #3) £ GL(p) x 0(r) x O(r') 
in ([21]) where in the notation of f )25l) and f )26|) 

/ 611 \ 



9x 



011 

&2i g?, 



and (X t 



92 > 



(in) T & T 



21^3 



#31 
2 4 i 





93 
a 33 

y 2 41 2 42 £43 (on 1 ) 7 / 

i/ere 6n G GL(p — r') ; 6 2 i G M r / iP _ r /(C) and a 33 G 0(m — 2p). Each Xij denotes an 
arbitrary entry so that g 2 is a matrix in 0(r). 

Proof. The action of g 3 G O(r') on 75 gives a matrix whose right bottom r' by r' sub- matrix 
is 03 and all entries above g 3 are zero. This implies that g\ G GL(p) must fix the subspace 
spanned by {f p - r '+i, ■ ■ ■ , f P }- Hence gi belongs to the maximal parabolic subgroup Q and 
the 622 entry in ( 12 5 p is g 3 . This gives the matrix gi in the lemma. Let E' denote the p 
by r matrix after deleting the last r' columns of E. The action of g\ on E' implies that 
02 must fixes row space V of E' and the subspace Vo spanned by {e' 1; . . . , e' }. Hence 
02 belongs to the maximal parabolic subspace fixing Vq. This implies that if (X~ 2 l ) T is of 
the form in f )26|) . then 012, ai3, ai4, a24, 034 are zero matrices. It fixes V implies that a 2 3 
and 032 are zero matrices too. It remains to identify the rest of the a^-'s and we will leave 
them to the reader. □ 

We refer to the matrices and f )2"B]) . We note that g 2 = {g 2 r ) T = [h^Y X g h T . 
Let N denote the unipotent subgroup generated by 231,241,243 in Lemma 17.41 We 

set N" = {(hr 1 ) T N h T G O(r)} and N' = { J 

Lemma [7.41 could be written as 

(27) gt = g 3 b u n and g 2 = g 3 bna 33 nn" 

where g 3 G O(r'), a u G GL(p - r'), a 33 G 0(m - 2p), n G iV and n" G N". 

Si = A((0(r') x GL{p - r')) x N) x 0(m - 2p)N" . 
Then T = r as an AO(r')-module, and the other factor subgroups of Si act trivially. 



G GL(p)}. Hence gi and g 2 in 
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Case (HI): r' > p > r. We begin with the matrix y/— IE, move the first r columns to the 
last r columns, and move the top r rows to the bottom r rows. Next we interchange the 
role of O(r) and O(r') and we are reduced to Case (II). We get the next lemma essentially 
from Lemma [7.41 by interchanging g 2 and g%. 

Lemma 7.5. With respect to certain coordinates, the stabilizer Sq consists of all (g 1; g 2 , g%) G 
GL(p) x O(r) x O(r') where 

Here bn G GL(p — r), b\ 2 G M r)P _ r (C) and a 22 G 0(m — 2p). Each Xij denotes an arbitrary 
entry so that g^ is a matrix in O(r'). □ 

Similarly, by interchange g 2 and g^ etc... in (|27p . we get 

Si = AO(r) x GLQo -r)iV 

in GL(p) x O(r) where iV is the unipotent subgroup generated by 621 in Lemma 17.51 
Let N denote the unipotent subgroup corresponding to Xi 2 , xu, x 2i in Lemma [7751 Then 
T = T o(m-2 P )W" wnere the subscript implies taking co-invariants. 

7.6. Definition of M\. We recall the character d p of P p in ([2]). We extend it to a 
character on P p x O(r) where it is trivial on 0(n — 2p)N p x O(r). We recall X{ = 
(Si x 0(n — 2p))N p in Section 15.61 We extend the representation T of Si to K[ where 
0(n — 2p)N p acts trivially. We will define a representation Mi of i^i in the following way 
according to Cases (I) to (Illb) listed in Theorem 11.61 

(I) If r, r' > p, then #J = K x and T = r°( r -*). We define Mi = d p 2 | Kl T. 

n — m 

(II) If r > p > r', then 2fJ = K x and T = t. We define Mi = d p 2 \ Kl T. 
(Ilia) Suppose r' > p > r and n > m. We set /i = IL ^ ; lp_ r .. By (J7J), contains 

m — 7i 

the i^ njr -type Co( n ) ® det ^ with multiplicity one. Let T = T Q ^ m _ 2p ^" ■ Let 

„ n-m 

Mi = IndtM p 2 |jf'T which is a representation of i^i ~ (0(r) x 0(ri — 2r))N r >. It 
is trivial on N r >. 

n — m 

We claim that Mi is either zero or det Q ^ ®Co(n-2r) as an O(r) x 0(n — 2r)- 

module. Indeed if r" is an irreducible 0(n — 2r)-submodule in Mi, then by the 
Frobenius reciprocity, the lowest weight of r" must be of the form 21 y n lp_ r + // 
where // is a weight of T. This forces // = (01 r , in Y !1 l p - r ) and r" = Co( n -2r)- 

n — m 

Hence Mi is generated by d p 2 tensoring the one dimensional extreme weight 
space of weight // in r. This proves our claim. 
(IIIb) Suppose r' > p > r and n < m. We set p = and r to be the trivial representation 
of O(r'). Then 7^ and T is the trivial representation of S\. We define 



/ (611 1 ) 1 " xi 2 (b lx ) T bJ 2 g 2 x u \ 



a 22 

V 






^24 


92 


X34 





hi J 
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Mi = Ind^Mp 2 \k[- It is a representation of K\ ~ (0(r) x 0(n — 2r))N r > which 

n — m m — n 

is trivial on iV r / and det ?x ®T ^ n _^ r on 0(r) x 0(n — 2r). 

7.7. We will now prove Theorem Ol We recall M = Gr$ and ikP = Gr $(//). Let 
be the corresponding coherent sheaf of M M . By Proposition II. 4[ A^ /i |o 1 is the K n ^ r - 
equivariant sheaf induced by M ei . Hence Theorem ll.6( ii) would follow immediately from 
part (i) and the definition of associated cycles. 

It remains to prove Theorem ll.fif i). For / e 1Z(Oi), we set /* = pr^(/) G 1Z(0). Let 
Q\ = (pr^d) n O. Then 0* D O = O by Proposition E^iii) and M{Ol) = C(O ) by 
Proposition 13. l( iii) . We have 

M"{O x ) = ^ cft (M^) / = ^ e ^Hbmo(ro(r,M / .) 

= Homo(r')(T-. tCO * M /*) = Hom 0(r ')(r, A^(C*)) 

(28) = Hom o(r0 (T, £((%)). 

In the above equality, we use the fact that /* is 0(r')-invariant and Homo( r ')( r ? — ) is an 
exact functor. We note that f l28l is valid for Cases (I) to (III). 
We recall that Oq is a K njr x 0(r')-orbit. Hence 

C(O ) = IndJ r xO(r) (d P ^kJ = IndJ r indJ;indS xO(r) (d p -^k ) 

= Ind^-IndJjCd^lKj *Ind| xO(r ' } C 50 ). 

In the above equalities, we have used K = (So x 0(n — 2p))N p and Sq C Si x O(r'). The 
induction *Ind means induction followed by trivial extension across 0(n — 2p)N p . Putting 
them into fl28|) gives 

(29) AT(e>i) = IndJ^nd^d^l^ Hom o(r0 (r, Ind| x ° (r,) C 5o )) = Indj£' r Mi. 

The second equality follows from the definition of T and M\ above. It is straightforward 
to check that above maps are all natural 7?.((9i)-module homomorphisms. 

Suppose UfCOu then Up C O* and Up n O = Up n C. It is clear that C(O ) ^ 0. 
We apply B = C(O ), the K n>r x 0(r')-orbit O and Z = = O to Proposition IO and 
Remark O and we get C(O ) f * = C(Up n O ). Hence £(£> )/* = C(U f .nO) = Mp and 

= Hom O ( rO (T,£(0o)/*) = Homo^j^M/.) = Mj? = Ai^(Uf). 

In particular for i 6 Oi we have A^ M (Oi) x = (M^) x where the subscript x denotes the 
localization with respect to the maximal ideal ra x at x. 

When x = e\, the complex vector space M ei = (M At ) ei /m ei (M M ) ei is generated by global 
sections. By ( |29|) . M ei is a subspace of M\. Since M ei is nonzero by Proposition II A\ Mi 
is also nonzero. 

It remains to show that .M M on 0\ is the i^ n;r -equivariant bundle induced by M±. This 
would follow immediately from Proposition 13.11 if we could show that the composite map 
M»(Oi) = B -> (M») x -> Mi at x = ei is surjective. In Cases (II), (Ilia) and (Illb), 
Mi is an irreducible representation of K\ so we apply Remark 13.21 It remains to check 
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Case (I). In this case, Mi is a semisimple representation of K\ (and also of Si) so it suffices 
to prove surjectivity onto any irreducible 0(p)-submodule M of Mj. Suppose T = 

m — n __ 

is an irreducible 0(p)-submodule of T = d p 2 Mi which is trivial on 0(n — 2p)N p . Then 

j^- I m — n I n — m 

B = Ind^^Mi will contain the i^ njr -type t q ,*\ P ® r o(r) which surjects onto d p 2 T . 
This completes the proof of Theorem II. 6( i). 

7.8. We will briefly discuss in Case (III) but not necessarily in Cases (Ilia) or (Illb). 

n-m 

We recall T = T 0(m _ 2p )]v" an d we define 7\ = Ind^|(d p 2 |^;T). 

First we claim that 7\ is a finite dimensional C-vector space. Indeed suppose Ai, A 2 , A3 
are i^-subquotients of T and satisfy an exact sequence 

-> Ai -> A 2 ->■ A 3 -> 0. 

By interpreting Ind^-Mx as global sections of a fG-equivariant sheaf on Ki/K[, we have 
a left exact sequence 

->■ Ind^Mi -> IndJ|A 2 -)■ Ind^|A 3 . 

Let ss(Ti) = ©jWj be the semi-simplification of T\ where each Wj is an irreducible 
representation of K[. In particular the unipotent part of K[ acts trivially. Then the 
above exact sequence implies that 

dimTi < ^dimlnd^V^-. 

j 

Using the explicit descriptions of K[ and Ki in Case (III), we conclude that each summand 
on the right hand side is finite dimensional. Hence dimTi is also finite dimensional. This 
proves our claim. 

Proposition 7.9. Let mi[Oi] be the associated cycle of &(fi) in Case (III)- Then mi < 
dim Ti . 

Proof. By replacing ~M X with Ti in Section Owe have M^(Oi) = Ind£"'Ti and M<*(Oi) x = 
(M^) x . This gives an injection of M ei = (M M ) ei /m ei (M M ) ei into 7\. Hence mi = 
dimM ei < dimTi and proves the proposition. □ 

Finally if r' > 2p, then is in the discrete series. By [Li2] . is a unitariz- 

able Harish-Chandra module with nonzero (qi, K® r )-cohomology. In particular one could 
compute its associated cycle by other means. 

7.10. We end this section with a conjecture of K. Nishiyama. The associated variety 
of the unitary lowest weight module L([x) is the closure of a single GL(p)-orbit. Let m' 
denote the multiplicity of the associate cycle. By |NOT] and [Yaj . m! = dim(r^) ^ ~ p \ 
By Theorem II .6[ m' is equal to the multiplicity mi of in Cases (I) and (II). This is 
an extension of Proposition 4.10 in |NZj for the dual pair Sp(p, M) x 0(n, r). This agrees 
with a conjecture of Nishiyama which states that the multiplicities of the associated cycles 
are preserved in theta lifts. However in Case (Illb) , m! = 1 but mi > 1. In Case (Ilia), 
mi = 1 but m' = 1 if and only if m = n. It seems that the stable range condition (i.e. 
2p < min(n, r) for our dual pair) is a necessary condition for the conjecture to hold. 
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8. UNIPOTENT REPRESENTATIONS 



8.1. First we will prove Proposition 11.71 We set x — e± — E + Ei £ Oi, stabilizer 
K x = Ki and \ = Mi. It is not difficult to check using the Frobenius reciprocity that 
Proposition 11.71 does not hold in Case (I). Hence we will assume Cases (II) and (III) for 
the rest of this section. 

Proposition 8.2. Suppose m > 2p and p > min(r',r). Let SOq = O — Oq denote the 
boundary of Oq in O. Then OOq has codimension at least two in O. 

Proof. By symmetry, we may assume that r > p > r'. By ( 1T7|) . dim O = dimK n)in — 
dimi^e = p(m + n — 1 — 2p). We assume the notation Af, Ao, O, Oq etc... in Section [5TT1 
First O — O is a union of 0(p', m; 0) where p' < p. By the same formula, dim 0(p', m; 0) = 
p'(m + n — 1 — 2p'). Since (p,n,m) is in stable range, dim O — dim 0(p',m; 0) = (p — 
p')[m + n — Ap — 1 + 2(p — p')] > 2. In order to prove the proposition, it suffices to show 
that O — Oq has codimension at least two in O. 

For X £ O, let C(X) denote its row space. Then C(X) is an isotropic subspace of C™ 
of dimension p. Let G(n,p)o denote the Grassmanian consisting of all isotropic subspaces 
of C n of dimension p. The map O — > G(n,p) given by X \- > C(X) commutes with the 
action of O(n). The fiber over C(E + E) is Af. Since Oq = (AdO(n))E + Af , it suffices to 
show that c > 2 where c is the codimension of — Ao in M . 

Let E(p, m) be the matrix in ffl2|) where n is replaced by m. We will define a p by m 
matrix E*. If m > 2p + 1 then there exists a column of zeros in E(p, m). Let E* denote 
the p by m matrix obtained from the matrix E(p, m) by removing this zero column and 
insert back as the last column. If m = 2p, then we set 



where I s is the s x s identity matrix. 

Lemma 8.3. The set M — Mq lies in the the closure of the GL(p) x O(r) x 0(r')-orbit 
Af* generated by E* . 

Proof. The proof is based on a simple idea but involves lengthy matrix manipulation. We 
will only give a sketch. For any X £ M p ^ r+r i, let X\ and X 2 be the first r columns and 
the last r' columns of X respectively. Pick X £ Af — (Ao U A"*). Using the action of 
GL(p) x O(r) x O(r') we may assume that X is a nice matrix with many zero entries. 
Based on the matrix X, we subdivide into a few cases. In each case, one shows that 
X = Um t ^oX(t) where {X(t) : t ^ 0} is a family of matrices X(t) £ Af — Ao such that 
X 2 (t)X 2 (t) T has higher rank than X 2 Xj . Note that Af* consists the element X £ Af — Afo 
such that X 2 X 2 has maximal possible rank among all X £ Af — Afo- Now the lemma 
follows by induction. □ 

We return to the proof of Proposition 18.21 where we want to show that c > 2. Let So 
and Sq be the stabilizers of E(p,m) and E* respectively in GL(p) x O(r) x O(r'). By 
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Lemma I8.3[ c = dim M — dim M* = dim Sq — dim Sq. By Lemma 17.41 
(30) dim Sq — — (p 2 — p + r 2 — r + (r') 2 — r') + p(p — r — r' + 1) . 

It remains to compute dimS'o- Let E\* denote the last r' rows of E* . 

First we consider the case m > 2p + l. Suppose g = (gi, g 2 , g 3 ) G GL(p) x O(r) x O(r') 
lies in S^. Then g stabilizes the row space of E 2 , i.e. g 3 G 0(r' — 1). Now the computation 
of So is exactly as that of Sq in Lemma 17.41 except that we have to reduce both m and r' 
by 1. Hence dimS'o is given by the right hand side of (130]) after reducing r' by 1. With 
these, we compute that c = dim Sq — dim S = 1 + p — r' > 2. 

Next we consider m = 2p. By choosing nice bases of C r and C r as in Section 17.31 one 
checks that X = (X 1 ,X 2 ,X 3 ) G gl(p) x so(r) x so(r') lies in the Lie algebra of Sq if and 
only if 

(a 22 ai2 b 13 \ /b 33 b X3 c \ /a u a l2 \ 

= a n , X 2 = a 22 -fcj, , X 3 = a 22 -a[ 2 

\o o 633/ \° -W \° - ai v 

where an G fll(l), a 22 G so(r' - 2), a 12 G M iy _ 2 , h 3 G M p _ r / + i )r /_ 2 , &33 G fll(p - r' + 1) 
and c G A 2 C p ~~ r ' +1 . Therefore dimS'o = \p 2 - 2pr' + \p + r' 2 - r' + 1. Hence c = 
dim Sq — dim So = 1 + p — r' > 2. This completes the proof Proposition 18.21 □ 

Lemma 8.4. Suppose OOq = O — Oq has codimension at least two in O . Then 

K n-m K y0 f_n n-m 

Ind£' m d p 2 = Indj£' r ° ( r) d p 2 . 
Proo/. Let K = K n , m and e = G 0. We define g : K -> by g(Jfe) = (Ad*A;)e. The 

j-r n — m n — m 

left hand side of the lemma is lnd£"' m d p 2 = C{0) = (R(K) <g) d p 2 ) Xl and the right 

hand side of the lemma is Ind^ rXO(r >d p 2 = C(O ) = (Uiq^Oo) ® d p 2 

It suffices to show that ft(lT) = ^g^Oo). We have K/K e ~ and q- l O /K e ~ C 
and Proposition 18.21 implies that g _1 £?o has codimension at least two in il". Now K is a 
nonsingular variety and hence it is normal. By Corollary 8.3 in pa], TZ{K) = 1Z(q~ l Oo). 
This proves the lemma. □ 

Proof of Proposition^^ By Lemma[E3J M M = Horner') (t'S £(0)) = Hom 0(r ')(r At , £(O ))- 
By (T2SD this is equal to M^Ox) = IndJ"' r M ei . We add that fl2g) is valid for Cases (II) 
and (HI). This proves Proposition 11.71 □ 

8.5. Proof of Corollary A 1.91 Suppose M ei = Mi is an admissible representation of K\. 
Then it is necessary that M\ is trivial on the connected component of the semi-simple 
part of K\. This implies that dim Mi = dimr = 1 in Case (II). 

Conversely we verify that M\ given in the corollary is an admissible representation 
of K\. In Case (Ilia) the connected component of K\ modulo its unipotent subgroup is 
SO(r) x SO(n — 2r). Since dim Mi = 1, the admissibility of M\ is trivially satisfied. 

Next we check (ii). We refer to {gi,g 2 ) G Si C K\ in Lemma 17.41 and the matrix bn 
in g x . Since M\ and x i n © are both one dimensional characters, it suffices to show that 
both characters agree on a diagonal matrix bn with diagonal entries {a, 1, ... , 1}. Using 
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the explicit description of Si in Lemma [73 and the fact that Ki = (Si x 0(n — 2p))N p , we 
check that both characters maps bn i-> a 11 ? 01 . This proves that Mi is admissible in (ii). 
The last assertion of the corollary is Proposition 11.71 This completes the proof of 



Special unipotent representations. We will briefly review special unipotent primitive 
ideals and representations in Chapter 12 in |V2j . Also see Section 2.2 in [T]. We will 
apply these to the representations in Corollary 11.91 

Let g = so(n + r, C). Let be the representation in Corollary 11.9( 11) where r > 
p > r', r + r' > 2p and dimr^ = 1. Its infinitesimal character corresponds to the 
weight oj = (q t i , Q n+r -2 P i Q2p-r'+2) under the Harish- Chandra parametrization |Loj . Here 
Qn — (y — 1) t — 2, . . .) denotes the half sum of the positive roots of so(N), and we 
insert or remove zeros from u if the string of numbers is too short or too long. The 
restriction of $(//) to (Q,K® r ) decomposes into a finite number of irreducible ($,K% r )- 
submodules. Let t?(//)° denote one of these irreducible (g, K® r )-submodules. Since Oi is 
also a K° r -orbit, $(/i)° also have associated variety 0\. 

We claim that the weight u also represents the infinitesimal character of ^(/i) as a 
(g, K® r )-module. Indeed we may have an ambiguity only if n + r is even where the 
infinitesimal character is either u or s(u). Here s is the involution induced by the outer 
automorphism of g. In this case r' is even and the weight u contains a zero in the string 
of numbers. Hence u and s(oS) represent the same infinitesimal character and proves our 
claim. 

Under the Kostant-Sekiguchi correspondence, 0\ generates a nilpotent SO(n + r, C)- 
orbit Ci in g*. The orbit Ci has the same Young diagram as that of Oi in Section [T73l less 
the plus and minus signs. Let Ji denote the primitive ideal of ^(/x) in U(q). The ideal Ji 
has a filtration {J ljS = U s (q) fl Ji : s G N}. By |V3j . Gr ( J x ) cuts out the variety C\ in g*. 

Let $ and i? be the roots and the root lattice of g. Let g v denote the simple Lie algebra 
with $ as coroots. In particular g v = so(n + r, C) if n + r is even and g v = sp( n+ 2~ 1 , C) if 
n + r is odd. We refer to Chapter 6 in [CM] on the order reversing map d (resp. <i v ) from 
the set of complex nilpotent orbits in g* (resp. (g v )*) to the complex nilpotent orbits in 
(g v )* (resp. g*). The orbit Ci is called special if it is in the image of d v and for a special 
orbit Ci, we have d y (d{Ci)) = C\. 

Suppose Ci is a special orbit. By the Jacobson-Morozov theorem, let {X v , H y ,Y V } be 
the s^-triple such that X v G d{Ci). We may assume that \H y lies in C ®z R. In this 
way defines an infinitesimal character of U(q) via the Harish- Chandra homomor- 
phism. If d(Ci) has Young diagram (oi, a 2 , . . . , a s ) then \H y = (g ai +i, Qa 2 +i, ■ ■ ■ , Qa s +i)- 

Suppose jH v gives the same infinitesimal character lo as that of ^(/i) . Let J denote 
the unique maximal primitive ideal of U(g) with infinitesimal character \H y . We will 
call J a special unipotent primitive ideal. By Corollary A3 in [B VJ , the variety cut out by 
Gr (J) in g* is the same as that of Gr ( Ji), namely C\. By Corollary 4.7 in [BKJ, Ji = J. 
We say that -#(yu)° is a special unipotent representation. 

Proposition 8.6. Suppose we are in Corollary \1.9{ ii), i.e. r + r' > 2p, r > p > r' and 
dimr M = 1. Then C% is a special orbit and i?(//)° is a special unipotent representation. 




□ 
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Proof. The orbit is special could be read off from page 100 in |CMj . Indeed d(C\) = 
(n + r — 2p — 1, 2p — r' + 1, r' — 1, 1) if r' > 1 and {n + r — 2p — 1, 2p, 1) if r' = 1. 
Furthermore d y d{Ci) = C\. We have ^H y = (g r ', Q n +r-2 P , Q2 P ~r'+2) = ^ which is the 
infinitesimal character of ^(/x) . The conclusion that $(fi)° is special unipotent follows 
from the discussion prior to the proposition. □ 

We apply the same considerations to Case (Ilia). Then C\ is a special orbit and \H y = 
(g n - r , Qr+i, Qr+i) if r is even and \H y = (g n - r , Qr+2, Qr) if t is odd. A similar proof as 
that of Proposition 18.61 gives the next proposition. 

Proposition 8.7. In Case (Ilia), is a special unipotent representation if and only 
if n = m = r + r' = 2p + e, fi = 0, 2p > r' > p and e G {1,2}. □ 
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